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SUMMARY 


This  paper  describes  the  Air  Force's  ongoing  job  performance  criterion  development  project. 
Specifically,  details  are  presented  on  the  background  and  conceptual  basis  for  this  work,  as  well 
as  the  job  performance  measurement  methodology  that  has  been  developed  to  date.  In  addition,  the 
strategy  being  used  to  Identify  a  measurement  system  across  clusters  of  specialties  Is 
discussed.  Finally,  ongoing  and  planned  research  and  development  efforts  which  support  this 
project  are  described. 
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PREFACE 

Approximately  4  years  have  passed  since  the  Initiation  of  a  long-tern  research  and 
development  program  focusing  on  job  performance  criterion  development.  This  pnper  Is 
Intended  as  an  Interim  description  of  the  project  -  Its  past,  present,  and  future.  An 
earlier  version  of  tE.ls  paper  was  presented  at  the  First  Annual  Conference  of  the 
Society  for  Industrial  and  Organizational  Psychology,  Chicago,  Illinois,  April  1986. 
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JOB  PERFORMANCE  MEASUREMENT:  A  SYSTEMATIC  PROGRAM 
OF  RESEARCH  AND  DEVELOPMENT 


I.  INTRODUCTION 

The  Air  Force  Human  Resources  Laboratory  (AFHRL)  Is  currently  conducting  a  large-scale  effort 
to  develop  a  measurement  technology  for  systematically  obtaining  job  performance  data.  Planning 
for  the  Mr  Force's  program  of  research  and  development  (RAD)  began  several  years  ago  as  the 
result  of  three  primary  requirements.  As  an  Initial  requirement,  operational  military  and 
civilian  program  managers  In  the  manpower,  personnel,  and  training  communities  asked  AFHRL  to 
develop  an  approach  for  measuring  job  performance  so  that  measures  could  be  used  to  assist  In  the 
evaluation  of  their  training  and  selection  programs.  In  addition,  the  appearance  In  1978  of  the 
Uniform  Guidelines  for  Employee  Selection  highlighted  the  Increasing  need  to  validate  civilian 
selection  procedures  against  job  performance  measures  (Casclo  &  Bernardln,  1981). 

Secondly,  the  manpower,  personnel,  and  training  research  community  needed  performance 
measures  to  serve  as  criteria  in  their  RAD  projects.  This  need  was  endorsed  when  an  AFHRl 
Research  Advisory  Panel  (composed  of  knowledgeable  scientists  from  academia  a>.d  Industry,  as  well 
as  peers  from  the  Army  and  Navy)  recommended  that  research  efforts  to  obtain  specific  performance 
criteria  be  consolidated  to  develop  a  job  performance  measurement  technology  composed  of  a  wide 
range  of  candidate  measurement  systems  to  serve  the  varied  projects'  needs. 

Plans  for  the  Air  Force  job  performance  measurement  effort  to  meet  these  requirements  were 
already  under  development  when  a  third  requirement  for  these  measures  came  with  the  Congressional 
mandate  to  test  the  feasibility  of  validating  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB)  against  job  performance  measures.  The  Impetus  for  this  mandate  occurred  In  1980,  when 
Congress  was  Informed  that  there  was  a  problem  with  the  normlng  of  the  ASVAB,  and  that  the 
Services  might  be  enlisting  a  greater  percentage  of  low-scoring  Individuals  than  was  previously 
thought  to  be  the  case.  (Normlng  Is  the  method  through  which  test  raw  scores  are  converted  Into 
percentile  scores  that  permit  them  to  be  compared  to  the  scores  of  a  defined  and  relevant 
population.  These  norms  allow  the  DoD  to  evaluate  Its  new  recruits  across  time  and  across  the 
Armed  Services.) 

As  a  result  of  the  normlng  Issue,  the  Assistant  Secretary  of  Defense  (Manpower,  Reserve 
Affairs,  and  Logistics)  directed  the  Military  Services  to  establish  an  RAD  program  to  link 
enlistment  standards  to  Job  performance.  In  1983,  Congress  tasked  the  Office  of  Assistant 
Secretary  of  Defense  to  provide  direct  oversight  for  joint-service  RAD  activities  addressing  Job 
performance  measurement.  In  addition,  DoD  requested  that  the  National  Academy  of  Sciences 
Impanel  a  Committee  on  the  Performance  of  Military  Personnel  (composed  of  nationally  recognized 
experts  in  scientific  and  technical  areas  related  to  performance  measurement).  This  committee 
provides  an  Independent,  technical  review  of  the  Joint-Service  project.  Thus,  the  operational 
requirements  and  the  legal  and  Congressional  mandates  provided  the  Impetus  to  planning  and 
obtaining  support  for  a  lengthy,  high-resource  RAD  effort. 

The  Air  Force  research  plan  has  been  developed  In  coordination  with  the  other  Services  to 
ensure  that  all  programs  are  complementary.  An  Integrated  set  of  cross-ServIce  and 
Service-specific  enlisted  specialties  has  been  designated  to  serve  as  the  Initial  test  bed  for 
the  development  of  job  performance  measurement  strategies.  Eight  Air  Force  specialties  (AFSs) 
were  selected  for  the  Air  Force  component  of  the  Joint-Service  project.  These  specialties  and 
their  aptitude  Index  areas  are  listed  In  Table  1. 

These  AFSs  cover  the  range  of  jobs  In  the  mechanical,  administrative,  general,  and 
electronics  aptitude  areas.  Further  developmental  efforts  are  planned  for  additional  enlisted 
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Table  1,  Air  Force  Specialties  Selected  for  Joint-Service  Effort 


Air  Force  specialty 

Specialty  code 

Aptitude  Index  area 

Jet  Engine  Mechanic 

Mechanical 

Ground  Radio  Operator 

492X1 

Administrative 

Air  Traffic  Control  Operator 

272X0 

General 

Avionic  Communications  Specialist 

328X0 

Electronics 

Aerospace  Ground  Equipment  Mechanic 

423X5 

Mechanical 

Personnel  Specialist 

732X0 

Administrative 

Aircrew  Life  Support  Specialist 

122X0 

General 

Precision  Measuring  Equipment  Specialist 

324X0 

Electronics 

AFSs  fro*  each  aptitude  area,  as  well  as  selected  officer  and  civilian  specialties.  In  addition, 
because  each  Service  has  chosen  a  unique  performance  measurement  orientation,  technologies 
developed  by  the  Air  Force  In  cross-ServIce  specialties  can  be  transferred  to  other  Services. 

The  overall  program  of  research  calls  for  the  development  of  Measurement  techniques  that  will 
allow  for  the  collectljn  of  valid,  accurate,  and  reliable  hands-on  job  performance  Information. 
These  measures  In  turn  will  be  used  as  benchmarks  against  which  surrogate  Indices  of  performance 
(less  expensive,  easler-to-admlnlster  Interview  tests,  and  performance  ratings)  will  be  evaluated 
as  substitutes  for  the  more  expensive,  labor-intensive,  hands-on  performance  measures.  In 
addition  to  newly  developed  performance  measures,  existing  measures  such  as  technical  training 
scores.  Airman  Performance  Report  ratings,  and  skill -level  advancement  Indices  will  also  be 
evaluated  as  possible  surrogate  measures. 

The  rationale  for  this  multi-faceted  approach  to  performance  measurement  Involves  cost 
considerations  (the  hands-on  measures  are  very  expensive  to  develop  and  administer),  but  It  Is 
also  based  on  the  hypothesis  that  the  various  methods  -  while  to  some  extent  overlapping  - 
measure  different  aspects  of  job  performance  with  differing  levels  of  accuracy.  (See  Kavanagh, 
Borman,  Hedge,  and  Gould  (1982)  for  a  detailed  discussion  of  this  approach.)  The  Air  Force 
strategy  for  this  multi-faceted  approach  will  be  discussed  In  more  detail  in  a  later  section  of 
this  paper. 


II.  CRITERION  DEVELOPMENT 

The  overall  purpose  of  the  RID  program  discussed  in  this  paper  Is  to  develop  and  evaluate  a 
strategy  for  using  measures  of  job  performance  for  validation  of  selection  and  classification 
procedures  and  for  evaluating  training  programs.  In  order  to  succeed  In  this  criterion 
development  effort,  the  Job  performance  measures  must  constitute  a  valid,  accurate  reflection  of 
an  Individual's  work  performance. 

A  variety  of  alternative  measures  are  available  for  assessing  performance  on  the  Job.  They 
range  from  subjective  to  objective  and  from  general  to  specific.  When  faced  with  the  choice  of 
which  criterion  to  select,  the  researcher  or  practitioner  typically  relies  on  several  Informal 
decision  rules: 

1.  cost  In  terms  of  time,  money,  safety,  or  mission  effects. 

2.  convenience  In  developing  or  obtaining  measures. 

3.  fidelity  or  accuracy  of  replicating  behaviors  relevant  to  the  job. 
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Unfortunately,  the  development  of  a  criterion  measure  Is  frequently  seen  as  a  secondary 
concern,  and  Decision  Rule  2  -  convenience  -  Is  frequently  applied.  The  result  Is  a  generic, 
packaged- to-pl ease  rating  fom  that  will  In  all  probability  satisfy  Decision  Rule  1  but  not 
Rule  3. 

When  the  chief  concern  of  researchers  and  practitioners  shifts  to  Decision  Rule  3,  and 
fidelity  becomes  the  overriding  concern,  measuring  a  sample  of  actual  work  presents  a  viable 
alternative  to  the  convenient  and  more  subjective  rating  form.  The  technique  of  work  sample 
testing  Involves  an  Individual's  performing  a  task  or  selected  set  of  tasks  relevant  to  his/her 
job  and  the  objective  scoring  of  performance  on  these  tasks.  The  value  of  the  work  sample 
methodology  lies  In  the  fidelity  with  which  the  selected  set  of  tasks  allows  measurement  of  an 
Incumbent's  job  proficiency.  Unfortunately,  this  can  also  be  a  weakness  of  the  technique.  Work 
sample  procedures  normally  Identify  critical  tasks,  discard  those  not  practically  measurable,  and 
then  simply  allow  the  remainder  to  become  the  selected  set  of  tasks  to  be  measured.  AFHRL's 
approach  to  work  sample  testing  Is  an  attempt  to  overcome  this  criterion  deficiency  problem. 


III.  AIR  FORCE  JOB  PERFORMANCE  MEASUREMENT  SYSTEM 

AFHRL's  approach  to  this  criterion  problem  calls  for  the  development  of  a  variety  of 
measurement  techniques.  The  most  detailed  method,  known  as  Walk-Through  Performance  Testing 
(WTPT) ,  combines  the  observation  of  hands-on  performance  and  Incumbent  Interview  testing  for  a 
set  of  representative  tasks.  In  addition,  task,  dimensional,  global,  and  Air  Force-wide  rating 
forms  have  been  developed  for  use  by  supervisors,  peers,  and  Incumbents.  All  four  sets  of  rating 
forms  require  evaluation  of  job  performance.  The  WTPT  and  rating  forms  In  combination  constitute 
the  Job  Performance  Measurement  System  (JPMS).  Ultimately,  performance  Information  provided  by 
such  a  system  should  prove  useful  for  evaluating  how  well  an  Individual  is  prepared  to  accomplish 
his/her  job  after  completion  of  the  various  stages  of  training  (e.g.,  technical  school,  field 
training  detachment,  or  on-the-job). 


Walk-Through  Performance  Testing 


For  the  Air  Force,  hands-on  testing  Is  a  particular  problem  because  of  the  complexity  and 
expense  Involved  In  performing  many  tasks.  For  example,  many  critical  tasks  cannot  be  measured 
by  hands-on  testing  because  these  tasks  tend  to  take  too  long  to  complete,  require  replacement  of 
expensive  parts,  or  risk  possible  damage  to  components.  AFHRL  has  developed  a  new  methodology  to 
deal  with  these  problems.  This  new  approach,  WTPT,  has  as  Its  foundation  the  work  sample 
philosophy  but  attempts  to  expand  the  measurement  of  critical  tasks  to  Include  those  tasks  not 
measured  by  hands-on  testing,  through  the  addition  of  an  Interview  testing  component  (Hedge, 
1984). 

The  hands-on  component  of  the  WTPT  resembles  a  traditional  hands-on  work  sample  test  designed 
to  measure  proficiency  on  a  critical  task.  For  example,  one  hands-on  task  may  require  an 
Incumbent  to  Install  a  starter  on  a  jet  engine.  On  the  first  page  of  the  examiner's  manual. 
Information  Is  provided  to  the  test  administrator  concerning:  testing  time;  required  tools, 
technical  orders,  and  job  guides;  pertinent  background  Information  and  necessary  engine 
configuration;  and  test  administrator  Instructions.  While  the  starter  Is  being  Installed,  the 
test  administrator  uses  a  checklist  to  Indicate  whether  steps  are  performed  correctly  or  not 
(e.g.,  lubricate  the  spline,  Index  the  position  of  the  staiver,  and  Install  the  locking  device). 
Finally,  a  5-polnt  rating  scale  allows  the  test  administrator  to  record  an  overall  rating  of 
proficiency  on  that  task. 
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Many  tasks  are  either  too  time-consuming,  too  costly,  or  too  dangerous  to  measure  by  hands-on 
testing.  Interview  testing  attempts  to  expand  the  content  domain  by  measuring  tasks  that  cannot 
be  measured  with  the  hands-on  method.  Interview  testing  requires  the  Incumbent  to  explain  the 
step-by-step  procedures  necessary  for  successful  completion  of  the  task.  This  allows  the  test 
administrator  to  assess  an  Incumbent's  proficiency-based  strengths  and  weaknesses  related  to  the 
performance  of  that  task.  For  example,  an  Interview  Item  may  test  an  Incumbent's  ability  to 
determine  the  source  of  high  oil  consumption.  Once  again,  on  the  first  page  of  the 
administrator's  manual,  pertinent  Information  Is  provided  to  the  test  administrator.  While  the 
Incumbent  Is  explaining  how  to  perform  the  task,  the  test  administrator  uses  a  checklist  to 
Indicate  whether  the  steps  necessary  for  successful  performance  are  correctly  described,  in 
addition,  a  5-polnt  overall  proficiency  rating  Is  recorded  by  the  administrator. 

The  interview  testing  is  conducted  at  the  work  site  In  a  "show-and-tell *  fashion  that  allows 
the  incumbent  to  ‘visually  and  verbally*  describe  how  a  step  Is  to  be  accomplished  (e.g.,  ‘that 
bolt  Is  to  be  turned  five  revolutions,*  or  ‘that  component  Is  to  be  lubricated  prior  to  being 
assembled*).  Thus,  Information  on  additional  tasks  can  be  collected  along  with  hands-on 
Information  to  provide  a  more  thorough  coverage  of  the  content  domain  and  a  more  accurate  picture 
of  an  Individual's  proficiency. 


Rating  Forms 


Information  learned  In  the  WTPT  development  process  was  used  to  develop  rating  forms  that 
range  In  detail  from  the  very  micro  to  the  very  macro.  This  range  provides  a  thorough  coverage 
of  the  performance  evaluation  continuum.  These  rating  forms  have  been  developed  as  either 
supplements  to  the  WTPT  process  or  as  substitutes  for  this  work  sample  approach.  Four  rating 
forms  -  task,  dimensional,  global,  and  Air  Force-wide  -  comprise  this  rating  system  across  three 
separate  rating  sources  (supervisor,  peer,  and  self). 

AFHRL's  approach  to  the  development  of  these  rating  forms  Involves  several  key  concepts.  One 
requirement  Is  scale  comparability  across  rating  forms.  All  rating  forms  are  constructed  using  a 
5-polnt,  adjectivally  anchored  rating  scale.  In  addition,  whenever  possible,  specific  behavioral 
descriptions  are  Included  to  provide  detailed  Information  to  assist  the  raters  In  making  accurate 
judgments.  Final  ,  a  minimal  competency  cutoff  Is  Incorporated  within  each  rating  scale,  such 
that  distinctions  can  be  made  between  those  airmen  who  meet  or  fall  to  meet  a  specified  level  of 
job  proficiency. 

Task  Rating  Form,  ihe  task  rating  form  provides  the  most  specific  rating  data.  Supervisors, 
peers,  and  Incumbents  are  asked  to  rate  proficiency  on  all  tasks  representative  of  the  job 
content  domain.  Only  the  5-polnt,  adjectivally  anchored  rating  scale  Is  utilized  for  this  rating 
form.  The  use  of  specific  behavioral  descriptors  for  each  task  (to  describe  the  different  levels 
of  proficiency)  is  considered  Impractical,  as  each  task  would  require  an  entire  page  to  complete. 

Dimensional  Rating  Form.  The  dimensional  rating  form  provides  the  second  most  specific 
rating  data.  Again,  supervisors,  peers,  and  Incumbents  are  asked  to  rate  the  technical 
proficiency  of  first-term  airmen  across  Important  areas  of  the  Job.  Potential  dimensions  are 
Identified  through  factor  analysis  of  occupational  survey  data,  and  scales  are  developed  using 
Input  from  Subject-Matter  Experts  (SMEs).  In  a  series  of  SME  workshops,  a  final  set  of 
dimensions  Is  Identified,  and  specific  behavioral  descriptions  are  developed  for  each  of  the  five 
scalar  values.  The  specific  _sscr1ptors  are  developed  using  a  behavioral  summary  statement  (BSS) 
approach  (Borman,  1979),  where  valid  SME-generated  behavioral  anchors  at  each  level  are  combined 
to  form  paragraph  descriptors  of  that  level. 
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Global  Rating  Form.  A  global  rating  for*  Is  developed  to  collect  supervisor,  peer,  and 
1ncutf>ent  overall  ratings  of  proficiency.  Two  1te*s  are  generated  to  cover  the  job  domain;  (a) 
technical  proficiency  and  (b)  Interpersonal  proficiency.  Once  again,  a  series  of  SHE  workshops 
Is  .sed  to  generate  specific  behavioral  descriptors  for  each  scalar  point  on  the  rating  form. 

Air  Force-Hide  Rating  Form.  The  Air  Force-wide  rating  form  Is  developed  to  be  representative 
of  all  specialties  In  the  Air  Force.  This  rating  for*  focuses  on  Important  performance  factors 
finalized  at  an  SME  workshop.  Because  of  this  Inter-specialty  focus,  SMCs  were  resource  managers 
working  In  the  Air  Force  Military  Personnel  Center  (AFMPC),  Asslgmaents  Section.  Resource 
managers  have  oversight  responsibilities  for  different  specialties,  and  their  combined  knowledge 
provided  the  details  for  constructing  a  BSS  scale  that  could  be  used  across  all  specialties. 

Additional  Information  Collected.  Another  Important  variable,  and  one  that  represents  a 
Joint-Service  commitment,  Is  job  experience.  Because  the  Air  Force  Is  the  lead  Service  for 
researching  the  job  experience/job  performance  link,  data  are  being  collected  on:  (a)  time  In 
service,  (b)  time  In  unit,  (c)  time  on  present  job,  (d)  number  of  times  each  task  has  been 
performed,  (e)  last  time  each  task  has  been  perforaed,  and  (f)  task-level  experl  *  ratings.  In 
this  way,  an  experience  composite  can  be  formed  and  related  to  performance  on  the  jc„. 

Over  and  above  this  Joint-Service  commitment,  AFHRL  Is  collecting  data  on  factors  related  to 
performance  (e.g.,  aptitude,  training,  situational  constraints,  motivation)  and  factors  related 
to  the  measurement  of  performance  (e.g.,  user  reaction  data  such  as  accuracy/acceptablHty  of 
measurement  Instruments,  clarity  of  Instructions). 


IV.  JPMS  DEVELOPMENT  STRATEGY 

Having  detailed  the  Job  Performance  Measurement  System  (JPMS),  It  Is  Important  to  explain 
AFHRL's  strategy  for  developing  and  testing  the  measurement  system  In  a  variety  of  specialties. 
The  primary  research  focus  for  this  effort  Is  the  application  of  the  job  performance  methodology 
within  enlisted  career  fields.  Given  that  the  RAD  effort  reported  In  this  paper  Is  driven  by  a 
multiple  measurement  philosophy,  with  the  objective  of  identifying  which  measure  or  combination 
of  measures  provides  the  optimal  criterion  for  validating  the  ASVAB,  a  more  thorough  explication 
of  this  strategy  Is  required.  In  addition,  it  should  be  kept  In  mind  that  the  ultimate  objective 
Is  Identification  of  surrogate  measures  that  can  be  substituted  for  the  more  expensive, 
time-consuming,  hands-on  procedure. 

Although  use  of  surrogates  is  the  desired  outcome  of  this  R&D  strategy,  decisions  must  be 
based  on  analysis  of  the  full  range  of  measurement  instruments  In  the  JPMS.  Figure  1  depicts  the 
JPMS  development  strategy  that  guides  the  Job  performance  measurement  project. 

The  Initial  phase  of  this  effort  Involves  JPMS  development  and  analysis  In  eight  AFSs.  The 
»  JPMS  methodology  was  first  developed  for  the  Jet  Engine  Mechanic  specialty  and  has  now  been 

applied  to  three  additional  specialties.  Application  to  the  final  four  AFSs  will  occur  during 
the  4th  quarter  of  FV86.  An  Important  pattern  to  note  with  regard  to  these  first  eight  AFSs  Is 
*  the  choice  of  specialties  from  the  four  aptitude  Index  (AI)  areas  of  the  ASVAB  (mechanical, 

administrative,  general,  and  electronics).  Development  and  testing  are  done  In  a  cyclical 
fashion  such  that  each  AI  Is  covered  before  the  process  is  repeated.  This  ensures  representative 
coverage  of  Air  Force  jobs.  This  cyclical  pattern  will  remain  constant  throughout  the  testing 
process. 
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APTITUDE 


JPMS  Development  Strategy 


Once  development  and  analysis  have  been  completed  on  all  eight  specialties,  decisions  will  be 
made  about  revision/reduction  of  the  JPMS  prior  to  application  of  the  system  to  eight  additional 
specialties.  These  decisions  nay  Involve  fine-tuning  of  JPMS  instruments  or  related  materials 
(e.g.,  revision  of  Instructions,  changes  In  rater  training,  changes  In  rating  form  layout  or 
format)  or  reduction  In  OPMS  Instruments  (e.g.,  after  eight  AFSs,  It  may  oe  determined  that 
self-ratings  provide  no  useful  Information  and  should  be  eliminated).  Durl'.g  the  second  phase. 
It  Is  still  desirable  to  Include  hands-on  testing  as  the  benchmark  for  svv  ogate  analysis.  In 
order  to  Identify  a  meaningful  pattern  of  surrogate  Instruments  ann  rating  sources  that 
adequately  cover  the  criterion  space,  and  to  provide  accurate  measurement  of  job  proficiency,  the 
hands-on  benchmark  should  be  Included  In  the  JPMS  as  long  as  it  is  feasible.  However,  due  to 
budgetary  constraints,  seme  reduction  In  the  depth  and  breadth  of  hands-on  testing  may  be 
reauired. 

Once  the  JPMS  has  been  tested  In  16  AFSs  (four  per  A1  area),  cross-specialty  results  will  be 
analyzed  in  order  to  identify  patterns  of  Instrument/source  superiority.  In  other  words,  an 
attempt  will  be  made  to  detect  clusters  of  specialties  that  are  defined  by  surrogate  Instrument 
commonality  (e.g.,  supervisor  task  ratings  may  be  superior  surrogate,  for  one  cluster,  whereas  a 
combination  of  peer  and  self  dimensional  ratings  may  be  more  useful  for  another  cluster  of 
specialties).  A  more  concrete  example  may  help  to  clarify  this  JPMS  development  strategy.  One 
possible  clustering  pattern  might  be  associated  with  aptitude  Index  areas.  That  Is,  perhaps  one 
g~oup  of  Instruments/sources  might  work  best  for  specialties  with  a  mechanical  AI,  whereas  a 
second  group  of  Instruments/sources  might  be  better  suited  for  administrative  specialties. 
However,  rather  than  hypothesizing  clusters  and  associated  Instruments,  these  patterns  will  be 
determined  empirically. 

Finally,  based  on  the  analyses  for  these  16  specialties,  appropriate  JPMS  components  will  be 
selected  and  applied  to  24  additional  AFSs  and  used  as  criteria  for  validating  the  ASVAB.  Once 
data  have  been  collected  and  analyzed  for  these  final  24  AFSs,  the  JPMS  methodology  will  have 
been  demonstrated  In  40  enlisted  specialties.  Sufficient  data  should  then  be  available  to 
Identify  the  necessary  measurement  Instruments  to  validate  the  enlisted  selection  test  for  any 
specialty. 


V.  APPLICATION  OF  JPMS  TECHNOLOGY  TO  TRAINING 
AND  OFFICER/CIVILIAN  SELECTION 

The  previous  section  of  this  paper  detailed  the  JPM5  development  strategy  for  validation  of 
the  ASVAB.  AFHRL's  plans  for  application  of  this  methodology  to  other  manpower,  personnel,  and 
training  requirements  will  now  be  discussed. 


Training  System  Evaluation 

Initial  JPMS  work  has  focused  on  enlisted  selection  system  validation,  as  depicted  In 
Figure  2.  However,  as  noted  earlier,  several  other  requirements  have  driven  AFHRL's  criterion 
development  effort.  Primary  among  these  additional  foci  Is  training  program  evaluation.  The  Air 
Force  conducts  extensive  training  of  new  enlistees  at  various  phases  of  their  enlistment 
(employment)  (l.e.,  technical  training  school,  field  detachment  training,  and  on-the-job  training 
( OJT ) ) .  Preliminary  criterion  work  will  concentrate  on  the  potential  of  the  JPMS  methodology  in 
evaluating  technical  school  training  and  OJT. 

Most  specialties  In  the  Air  Force  provide  Initial  training  once  recruits  have  been  classified 
Into  a  career  field.  After  completing  basic  military  training,  recruits  are  sent  to  a  technical 
training  school  for  their  specialty  for  a  specified  period  of  time  (length  of  training  varies  by 
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AFS).  Training  Is  given  In  blocks  of  Instruction,  and  learning  Is  evaluated  with  a 
paper-and-pencl 1  test.  Success  In  technical  training  Is  then  assessed  by  averaging  scores  across 
blocks  of  Instruction.  The  JPMS  methodology  can  be  utilized  to  evaluate  aore  accurately 
performance  In  technical  school. 

Once  technical  school  graduates  have  been  placed  In  their  first  job  assignment,  the  JPMS 
■ethodology  can  then  be  applied  to  assess  their  proficiency  on  the  job.  Once  again,  Individual 
strengths  and  weaknesses  can  be  Identified,  fed  back  to  the  Individuals,  and  fed  back  to  the 
training  coamwnlty.  This  Information  provides  an  opportunity  for  both  technical  school  and  OJT 
trainers  to  assess  the  strengths  and  weaknesses  of  their  training  programs.  Work  In  the  training 
arena  Is  In  the  planning  stages  and  due  to  begin  In  FY87. 


Offlcer/Clvllltn  Selection  System  Validation 


The  JPKS  methodology  will  also  be  transitioned  to  the  officer  and  civilian  selection 
domains.  Just  as  job  performance  measures  are  needed  to  validate  the  enlisted  selection  system, 
a  similar  approach  may  be  applied  to  the  officer  ranks;  however.  It  1$  expected  that  this 
transition  will  not  occur  without  some  changes  to  the  JPMS.  Although  the  theory  and  methodology 
will  still  be  applicable,  certain  refinements  to  the  JPMS  will  probably  be  required.  Given  the 
nature  of  officer  work,  It  Is  anticipated  that  the  NTPT  component  of  the  JPMS  may  more  closely 
resemble  an  assessment  center  design.  Development  of  officer  performance  mearures  Is  planned  for 
FY87. 


Similarly,  the  Air  Force's  civilian  selection  system  Is  In  need  of  job  performance  measures 
to  serve  as  criteria  for  validation.  A  feasibility  study  to  assess  the  usefulness  of  the  JPMS 
methodology  for  measuring  the  job  performance  of  Air  Force  civilians  Is  planned  for  the  4th 
quarter  of  FY86.  Development  and  Implementation  of  job  performance  measures  In  one  civilian  job 
are  scheduled  for  FY87. 

Extending  the  JPMS  methodology  to  officer  and  civilian  jobs  is  dependent  on  continued  funding 
at  current  project  funding  levels. 


VI.  INTEGRATION  OF  PRIMARY  AND  SUPPORTIVE  RAD 

In  addition  to  the  JPMS  development  work  which  has  been  underway  since  FY84,  a  number  of 
ongoing  and  planned  RAD  efforts  support  this  primary  work  effort.  Figure  3  describes  visually 
how  the  results  of  these  efforts  are  being  Integrated  Into  the  validation  project.  Rather  than 
describing  each  research  component  and  how  it  supports  the  JPMS,  several  example  studies  will  be 
discussed. 

*  One  RAD  effort  currently  in  progress  concerns  rater  training,  as  shown  In  Figure  3.  This 

effort  has  been  designed  to  address  training  Issues  for  walk-through  test  administrators, 
supervisors,  peers,  and  Incumbents  who  will  be  completing  the  four  types  of  rating  forms  In  the 
JPMS.  The  Interaction  of  type  of  training  and  type  of  rating  form  Is  being  Investigated,  as  are 
the  four  major  components  of  the  behavior  modeling  paradigm  (Sorcher  A  Goldstein,  1972).  The 
overall  test  bed  for  examining  these  Issues  employs  an  assessment  center  methodology,  whereby 
both  obseryatl on/recording  components  (as  required  for  NTPT)  and  ratings  (as  required  for  rating 
forms  In  the  JPMS)  are  Included.  In  addition,  videotape  technology  Is  being  used  as  a  mechanism 
for  gathering  accuracy  data.  Finally,  the  development  of  an  assessment  center  methodology  should 
prove  helpful  when  developing  the  NTPT  for  officer  Jobs,  as  well  ns  certain  other  enlisted 
specialties  (e.g..  Personnel). 
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Figure  3,  Integration  of  Primary  and  Supportive  RAO. 


10 


A  second  ongoing  effort  Involves  the  development  of  a  minimal  competency  methodology  for 
establishing  cutoff  scores  for  the  WTPT.  Presently,  this  methodology  Is  being  applied  to  the  Jet 
Engine  Mechanic  specialty,  and  cutoff  scores  have  been  derived.  This  research  should  prove 
helpful  In  two  main  areas.  First,  Information  about  the  percentage  of  job  Incunfcents  tested  who 
are  considered  minimally  competent  In  a  specialty  may  be  useful  data  for  assisting  the  enlistment 
standard-setting  process.  In  addition,  the  training  community  should  benefit  from  this  knowledge 
when  evaluating  training  effectiveness.  A  brief  explanation  of  each  RAD  effort  underway  or 
planned  can  be  found  In  Table  2. 


Table  2.  Sui 

■ary  of  RAO  Efforts 

Title 

Completion 
schedul e 

Brief  descrlptlon/major  variables 

Job  Performance  Meta-Analysis 

FY85 

(1)  Supervisor,  peer,  self  ratings  of  job 
performance  data;  (2)  Multi  tralt/multlmethod 
(MTMM)  analyses  of  job  performance  data 

WTPT  Administrator 

FY85 

Contractor,  active  duty  (on-slte/off-slte) • 

rater/ratee  familiarity 

Rating  Form  Accuracy 

FY86 

Purpose  of  appraisal;  rating  form  specificity; 
mode  and  level  of  Instructional  detail;  accuracy 
paradigm;  acceptability,  motivation,  confidence, 
trust 

Structural  Equation  Modeling 

FY86 

Confirmatory  factor  analysis  of  MTMM  JPMS  rating 
data;  covariance  structure  analysis  of  Jet  Engine 
JPMS  data 

Training  Effectiveness  Overview 

FY86 

Air  Force  training  system;  use  if  JPMS  for 
training  evaluation 

Rater  Training 

FY86 

Type  of  training;  Interaction  with  rating  firms; 
behavior  modeling;  videotape  accuracy;  assasment 
center  test  bed 

WTPT  Minimal  Competency 

FY86 

Cut-score  methodology;  application  to  Jet  Engine 
Mechanic 

General  Izablllty  Theory 

FY86 

Application  to  JPMS  development,  training, 
administration 

Time  to  Job  Proficiency 

FY87 

Time-based  supervisor  rating  forms;  aotltu-e/ 
experience  performance  model 

WTPT  Accuracy 

FY87 

Error/accuracy  Interrelationship;  rater  ‘r.di  victual 
differences 

JPMS  Technology  to  Training 

FY92 

Transition-  of  JPMS  methodology  to  evaluate 
training  effectiveness 

JPMS  Technology  to  Officers 

FY92 

Transition  of  JPMS  methodology  to  officer  jobs 

JPMS  Technology  to  Civilians 

FY92 

Transition  of  JPMS  methodology  to  validate  the  ci¬ 
vilian  Promotion,  Placement,  and  Referral  System 
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VII.  SUMMARY 


If  Air  Force  use  of  job  performance  measures  In  validating  selection  procedures  proves 
feasible,  a  specialty-specific  performance  measurement  database  can  be  kept  current  and  used 
routinely  to  validate  these  selection  tests  and  procedures.  The  performance  measurement 
technology  and  resulting  data  can  then  be  used  by:  (a)  the  manpower,  personnel,  and  training  RAO 
community  to  evaluate  their  RAD  products,  (b)  the  operational  manpower  and  personnel  community  to 
evaluate  the  Impact  of  personnel  policies  and  procedures,  and  (c)  the  operational  training 
community  to  develop  and  evaluate  training. 

The  Air  Force's  most  promising  application  of  this  technology  outslae  of  the  enlisted 
selection  system  project  Is  In  training  evaluation.  The  training  Interface  has  been  specifically 
reguested  by  Headquarters  Air  Training  Command,  and  Initial  Investigation  of  the  applicability  of 
this  technology  to  training  1$  underway.  The  results  of  the  Initial  research  will  serve  as  the 
guide  for  further  RAD  In  the  training  area.  If  funding  permits,  plans  call  for  continuation  of 
the  training  applications  through  FY92. 
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